{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "90c6730f-5d76-450b-9788-ec883d024f57",
   "metadata": {},
   "source": [
    "# Hugging Face Transformers 微调训练入门\n",
    "\n",
    "本示例将介绍基于 Transformers 实现模型微调训练的主要流程，包括：\n",
    "- 数据集下载\n",
    "- 数据预处理\n",
    "- 训练超参数配置\n",
    "- 训练评估指标设置\n",
    "- 训练器基本介绍\n",
    "- 实战训练\n",
    "- 模型保存"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0b1e12-1921-4438-8d5d-9760a629dcfe",
   "metadata": {},
   "source": [
    "## YelpReviewFull 数据集\n",
    "\n",
    "**Hugging Face 数据集：[ YelpReviewFull ](https://huggingface.co/datasets/yelp_review_full)**\n",
    "\n",
    "### 数据集摘要\n",
    "\n",
    "Yelp评论数据集包括来自Yelp的评论。它是从Yelp Dataset Challenge 2015数据中提取的。\n",
    "\n",
    "### 支持的任务和排行榜\n",
    "文本分类、情感分类：该数据集主要用于文本分类：给定文本，预测情感。\n",
    "\n",
    "### 语言\n",
    "这些评论主要以英语编写。\n",
    "\n",
    "### 数据集结构\n",
    "\n",
    "#### 数据实例\n",
    "一个典型的数据点包括文本和相应的标签。\n",
    "\n",
    "来自YelpReviewFull测试集的示例如下：\n",
    "\n",
    "```json\n",
    "{\n",
    "    'label': 0,\n",
    "    'text': 'I got \\'new\\' tires from them and within two weeks got a flat. I took my car to a local mechanic to see if i could get the hole patched, but they said the reason I had a flat was because the previous patch had blown - WAIT, WHAT? I just got the tire and never needed to have it patched? This was supposed to be a new tire. \\\\nI took the tire over to Flynn\\'s and they told me that someone punctured my tire, then tried to patch it. So there are resentful tire slashers? I find that very unlikely. After arguing with the guy and telling him that his logic was far fetched he said he\\'d give me a new tire \\\\\"this time\\\\\". \\\\nI will never go back to Flynn\\'s b/c of the way this guy treated me and the simple fact that they gave me a used tire!'\n",
    "}\n",
    "```\n",
    "\n",
    "#### 数据字段\n",
    "\n",
    "- 'text': 评论文本使用双引号（\"）转义，任何内部双引号都通过2个双引号（\"\"）转义。换行符使用反斜杠后跟一个 \"n\" 字符转义，即 \"\\n\"。\n",
    "- 'label': 对应于评论的分数（介于1和5之间）。\n",
    "\n",
    "#### 数据拆分\n",
    "\n",
    "Yelp评论完整星级数据集是通过随机选取每个1到5星评论的130,000个训练样本和10,000个测试样本构建的。总共有650,000个训练样本和50,000个测试样本。\n",
    "\n",
    "## 下载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bbf72d6c-7ea5-4ee1-969a-c5060b9cb2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset(\"yelp_review_full\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ec6fc806-1395-42dd-8121-a6e98a95cf01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['label', 'text'],\n",
       "        num_rows: 650000\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['label', 'text'],\n",
       "        num_rows: 50000\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c94ad529-1604-48bd-8c8d-aa2f3bca6200",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': 2,\n",
       " 'text': \"As far as Starbucks go, this is a pretty nice one.  The baristas are friendly and while I was here, a lot of regulars must have come in, because they bantered away with almost everyone.  The bathroom was clean and well maintained and the trash wasn't overflowing in the canisters around the store.  The pastries looked fresh, but I didn't partake.  The noise level was also at a nice working level - not too loud, music just barely audible.\\\\n\\\\nI do wish there was more seating.  It is nice that this location has a counter at the end of the bar for sole workers, but it doesn't replace more tables.  I'm sure this isn't as much of a problem in the summer when there's the space outside.\\\\n\\\\nThere was a treat receipt promo going on, but the barista didn't tell me about it, which I found odd.  Usually when they have promos like that going on, they ask everyone if they want their receipt to come back later in the day to claim whatever the offer is.  Today it was one of their new pastries for $1, I know in the summer they do $2 grande iced drinks with that morning's receipt.\\\\n\\\\nOverall, nice working or socializing environment.  Very friendly and inviting.  It's what I've come to expect from Starbucks, so points for consistency.\"}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[\"train\"][111]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6dc45997-e391-456f-b0b9-d3193b0f6a9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import pandas as pd\n",
    "import datasets\n",
    "from IPython.display import display, HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9e2ecebb-d5d1-456d-967c-842a79fdd622",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_random_elements(dataset, num_examples=10):\n",
    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
    "    picks = []\n",
    "    for _ in range(num_examples):\n",
    "        pick = random.randint(0, len(dataset)-1)\n",
    "        while pick in picks:\n",
    "            pick = random.randint(0, len(dataset)-1)\n",
    "        picks.append(pick)\n",
    "    \n",
    "    df = pd.DataFrame(dataset[picks])\n",
    "    for column, typ in dataset.features.items():\n",
    "        if isinstance(typ, datasets.ClassLabel):\n",
    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
    "    display(HTML(df.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1af560b6-7d21-499e-9b82-114be371a98a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2 star</td>\n",
       "      <td>I stopped in for dinner last night, craving their grits... the only thing that was any good during this visit were the fried eggs.  I ordered the High Flyer again, my usual order at both locations, and it was a wreck.  \\n\\nService was friendly and attentive, as usual.  On a Friday night, there were two other tables seated, and they were finishing up when I sat down.  Both Biscuits refuse to fully air condition the restaurant, so it was as tropical as the table cloths.  \\n\\nThe food:\\nThe eggs, cooked to over-well, were perfect.  \\nThe chicken sausage was over cooked and almost crunchy.  So overly seasoned that I woke up at 1 am and it felt like my chest was on fire from indigestion.\\nThe biscuit - well, I stated before how I feel about their biscuits.  Too heavy and really just a way to get the delicious apple cranberry butter into my pie hole. But at least it wasn't burned.\\nI substituted a buttermilk pancake for the oatmeal pancake and it was burned AGAIN.  It's a pancake - if you burn it, throw it out and give me another one - it's not a steak, it's batter that probably costs less than a nickel.  \\nThe grits - my favorite grits - grits that make my mouth water every time I drive by this or the Park Rd location - were AWFUL.  Overly cheesed and overly salty, clumps of grits were stuck together - BAD.  \\n\\nAs my Pawpaw said, during his final days, after spending the weekend with a family member that will make you wish death would come soon, \\\"I ain't goin' back\\\".</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2 star</td>\n",
       "      <td>This was my first visit to this location for Sunday Champagne Brunch.  It was located in the Cannery Casino on E Craig Road in Las Vegas.  We had free buffet coupons from spinning the wheel on Web Pass.\\n\\nThe stations were not labeled, so you had to walk up to see what there was.  At the Cannery Row Buffet on Boulder Highway, the stations were clearly marked with big signs.  They had American, Asian, Italian, Mexican, 2 large Salad Bars, and Desserts. The server took our drink order and brought us drinks and cleared our empty plates.  We were not offered champagne.\\n\\nFor brunch there was a made to order omlette station.  This was probably the best and freshest feature.  There was no Egg Benedict.  The bacon was limp and greasy.  The hash browns were shredded and burned.  The sausage links had a lot of filler.  The gravy for the Biscuits and Gravy had been sitting there for a while.\\n\\nThe Italian meatballs were large, but had a lot of filler.  The tortellini was meh.\\n\\nThe taquitos were ok if you like the kind that come in the freezer section.  The cheese sauce was runny.  There were tamales wrapped in plastic wrap that I did not try. \\n\\nThe potstickers were dried out and fried.  The Broccoli Beef was okay, but had thin strips of beef.   The egg rolls were tastless.\\n\\nThe pepperoni pizza was okay when it was fresh and hot.  There was cheese bread.\\n\\nThe salad bar had lettuce, lots of toppings, pasta salad, ambrosia, assorted fruit, rolls, etc.  This was much nicer than the Cannery Row Buffet on Boulder Highway.\\n\\nAgain, the desserts were the best part of the meal.  I had to ask for the Coconut Cream Pie.  Yumm!  The Apple Custard Tart was good (I had 2 slices).  There was Boston Cream Pie, Carrot Cake, Cheesecake, Tiramisu, wedding cookies, soft-serve ice cream, and plenty of sugar-free desserts.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1 star</td>\n",
       "      <td>The road is paved now so that's great news. Unfortunately it's very over priced.  When you finally get there, you still have to get on another shuttle bus to get to the skywalk.  The staff was VERY unprofessional. You are not allowed to take your own camera so they can charge you a crazy amount to take pictures themselves.  While sightseeing, we were constantly being asked to move so the staff could take pictures of other tourists. They try to usher you off the Skywalk so they can get in the next group. \\n\\nGreat views but terrible service. Not worth the time, effort or price. I would like 2/3 of my money back please.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4 stars</td>\n",
       "      <td>One of the best pizzas I've had in Charlotte. They also sell pizza by the slice which is a plus.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2 star</td>\n",
       "      <td>I try cupcake shops everywhere I go and was excited to stop here. They're located inside a fantastic building with other quaint little shops surrounding it. We picked up  a dozen minis ( that's all they sell at this location) of pre selected pre packaged flavors. There were 2 each of six different flavors cupcakes in the package  .... But it might as well have been 3 flavors! Unless you have a chocolate or red Velvet all the cake is the same! Boringgggggggg! The Snickerdoodle, birthday and vanilla all taste the same the only difference is the sprinkles on top. Such a disappointment . I will say they were very moist which is why I gave 2 stars instead of 1.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2 star</td>\n",
       "      <td>So first off, an orange table on a sign on a pillar facing the opposite direction of the entrance into the park from the parking garage is not a great way to show people that your business is in fact right there.  We wandered to the left and right of the prominent Jewel of the Crown and finally had to call the brusque busy man who half thought we were at the Fashion Square though I started the conversation with where exactly in the mall are you because we're in front of AZ88.  \\n\\nWe finally figure out the door which was also covered by a huge 7 foot tall man watching another man on a ladder screw in a bracket.  Both men have their backs to us and are completely blocking the door.  We walk up behind them and watch the man screw in the bracket and finally the tall man moves aside just enough for me to get to the door.  Guess they weren't expecting customers.\\n\\nThe food itself isn't awful.  The bloody mary is tiny for a $7 cocktail and both burger and sandwich were a bit overpriced for what they were.  They both came with some potato chips and a pickle and came in a little over $10 each.  The burger was actually decent, though it was a little rare for a medium.  The flavor of the meat was actually quite nice.  The sandwich was ok, though it almost seemed like sliced deli meat pan fried (not a breast).  \\n\\nThey're shutting down and renewing themselves as some other establishment.  Hopefully they'll have a sign up by then.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5 stars</td>\n",
       "      <td>YAY!! Rachel C at EWC is awesome, thank you sooooo much for her. I \\\"go to brazil,\\\" with  Rachel every 3-4 weeks. She has always been on time, very nice, and professional. She makes me feel comfortable, and is very cean, never a hair missed. :)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3 stars</td>\n",
       "      <td>Some good sights seeing while at the stadium. Good view from where ever you sit. Had a great time inside and out during the recent Sevens Rugby tournament festivities. But it is old and outdated. OK at best. They should level it and put just off the strip.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1 star</td>\n",
       "      <td>Shared a room with a friend who flew into town for business and didn't want to stay alone. I was confused as to why she would chose the Artisan but her company arranged all her travel plans. She asked for my company for the night after reading some reviews.\\n\\nThe one star is for the eclectic and artsy decor, which I love, because it was a refreshing change from what we normally fund in Vegas. The staff was pretty cool too.\\n\\nBUT.... I will never book my own stay here or come to hang at the bar. The drinks were not good and way over priced. The carpet in my friend's suite was so dirty the bottom of my yellow flip flops turned black, only I didn't realize this until I checked them because I had taken them off and the carpet felt sticky on my bare feet. There was an ashtray sitting outside the window, which was full, and trash strewn across the roof was our view. Bathroom fixtures were broken and the radio/dvd player didn't work.\\n\\nEVEN WORSE was the fact that we came out of the room TWICE to a creeper with his ear plastered to our door. What was supposed to be a chill evening evening turned into us escaping to the Las Vegas strip to relax.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4 stars</td>\n",
       "      <td>On a perfectly mild sunny day in vegas, it was a great choice to eat on the patio at Mon Ami Gabi overlooking Las Vegas Blvd. As soon as we entered the restaurant from the casino entrance, from the dark interior, it was like stepping into a whole different world once we got seated on the patio. \\nUsually when servers rattle off the specials of the day, it goes out the other ear for me. But after looking over the menu and half heartedly deciding on the spinach and gruyere omlette, I heard the server mention Bacon and Spinach Eggs Benedict and i was sold! Quiche Lorraine with bacon http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=4Ua39HREZlvlYbEf3EP_ew for the dude and my friend got the dish i was thinking of originally http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=5luau0SKUmjSVfXBalxsMQ. We all agreed that my dish ended up being the best: http://www.yelp.com/biz_photos/4JNXUYY8wbaaDmk3BPzlWw?select=T3jPSmE56xcETgTqjegttw.\\nThe quiche had a nice crust, but was kind of monotonous. (i love breakfast sides and this had none). The spinach omelette was ok- i enjoyed my omelette at Max Brenner much more. \\nI've only had eggs benedict a few times elsewhere and i have to say this one was very yummy- the eggs were perfectly runny and the bacon (i believe it was applewood) was nice and crispy. \\nWe were too sweets wasted this time to order the pancakes, french toast etc, so next time i would like to try those or the steak dishes with a side of their onion soup!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(dataset[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9df7cd0-23cd-458f-b2b5-f025c3b9fe62",
   "metadata": {},
   "source": [
    "## 预处理数据\n",
    "\n",
    "下载数据集到本地后，使用 Tokenizer 来处理文本，对于长度不等的输入数据，可以使用填充（padding）和截断（truncation）策略来处理。\n",
    "\n",
    "Datasets 的 `map` 方法，支持一次性在整个数据集上应用预处理函数。\n",
    "\n",
    "下面使用填充到最大长度的策略，处理整个数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8bf2b342-e1dd-4ab6-ad57-28eb2513ae38",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
    "\n",
    "\n",
    "def tokenize_function(examples):\n",
    "    return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
    "\n",
    "\n",
    "tokenized_datasets = dataset.map(tokenize_function, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "47a415a8-cd15-4a8c-851b-9b4740ef8271",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2 star</td>\n",
       "      <td>Here's the deal: Long Lines - Overpriced - Good Service - Average Food.\\n\\nI would LOVE IT if Steve would actually eat in his own buffet so he could see what it has become. When we were here years past the quality of the food was so much better (but the service was lacking). Now it's the opposite-- the service is good but you can really notice the decline in the food (cuts of meat, etc). And, two small glasses of wine will set you back a good $25. \\n\\nI know it's the strip and I don't mind paying $40 a person IF the food is good (you'd pay that in a decent restaurant anywhere else). My problem is that the Wynn prides itself on quality... \\n\\nActually, to be honest I've decided I now officially hate buffets. I can't stand watching people stuffing themselves and coming back to their table with plates piled high with 50+ crab legs. Not to mention the amount of food that is wasted is ridiculous. \\n\\nPeople, what have we become? \\n\\nBTW, if you want a good buffet, skip the Wynn and head over to the new one at the Cosmopolitan Hotel.</td>\n",
       "      <td>[101, 3446, 112, 188, 1103, 2239, 131, 3261, 12058, 118, 3278, 1643, 10835, 1181, 118, 2750, 2516, 118, 18098, 6702, 119, 165, 183, 165, 183, 2240, 1156, 149, 2346, 17145, 9686, 1191, 3036, 1156, 2140, 3940, 1107, 1117, 1319, 171, 9435, 2105, 1177, 1119, 1180, 1267, 1184, 1122, 1144, 1561, 119, 1332, 1195, 1127, 1303, 1201, 1763, 1103, 3068, 1104, 1103, 2094, 1108, 1177, 1277, 1618, 113, 1133, 1103, 1555, 1108, 11744, 114, 119, 1986, 1122, 112, 188, 1103, 3714, 118, 118, 1103, 1555, 1110, 1363, 1133, 1128, 1169, 1541, 4430, 1103, 6246, 1107, 1103, 2094, 113, 7484, 1104, 6092, ...]</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(tokenized_datasets[\"train\"], num_examples=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c33d153-f729-4f04-972c-a764c1cbbb8b",
   "metadata": {},
   "source": [
    "### 数据抽样\n",
    "\n",
    "使用 1000 个数据样本，在 BERT 上演示小规模训练（基于 Pytorch Trainer）\n",
    "\n",
    "`shuffle()`函数会随机重新排列列的值。如果您希望对用于洗牌数据集的算法有更多控制，可以在此函数中指定generator参数来使用不同的numpy.random.Generator。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a17317d8-3c6a-467f-843d-87491f600db1",
   "metadata": {},
   "outputs": [],
   "source": [
    "small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
    "small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b65d63-2d3a-4a56-bc31-6e88a29e9dec",
   "metadata": {},
   "source": [
    "## 微调训练配置\n",
    "\n",
    "### 加载 BERT 模型\n",
    "\n",
    "警告通知我们正在丢弃一些权重（`vocab_transform` 和 `vocab_layer_norm` 层），并随机初始化其他一些权重（`pre_classifier` 和 `classifier` 层）。在微调模型情况下是绝对正常的，因为我们正在删除用于预训练模型的掩码语言建模任务的头部，并用一个新的头部替换它，对于这个新头部，我们没有预训练的权重，所以库会警告我们在用它进行推理之前应该对这个模型进行微调，而这正是我们要做的事情。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4d2af4df-abd4-4a4b-94b6-b0e7375304ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b44014df-b52c-4c72-9e9f-54424725a473",
   "metadata": {},
   "source": [
    "### 训练超参数（TrainingArguments）\n",
    "\n",
    "完整配置参数与默认值：https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments\n",
    "\n",
    "源代码定义：https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/training_args.py#L161\n",
    "\n",
    "**最重要配置：模型权重保存路径(output_dir)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "98c01d5c-de72-4ff0-b11d-e07ac5346888",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "model_dir = \"models/bert-base-cased-finetune-yelp\"\n",
    "\n",
    "# logging_steps 默认值为500，根据我们的训练数据和步长，将其设置为100\n",
    "training_args = TrainingArguments(output_dir=model_dir,\n",
    "                                  per_device_train_batch_size=16,\n",
    "                                  num_train_epochs=5,\n",
    "                                  logging_steps=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0ce03480-3aaa-48ea-a0c6-a177b8d8e34f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainingArguments(\n",
      "_n_gpu=1,\n",
      "adafactor=False,\n",
      "adam_beta1=0.9,\n",
      "adam_beta2=0.999,\n",
      "adam_epsilon=1e-08,\n",
      "auto_find_batch_size=False,\n",
      "bf16=False,\n",
      "bf16_full_eval=False,\n",
      "data_seed=None,\n",
      "dataloader_drop_last=False,\n",
      "dataloader_num_workers=0,\n",
      "dataloader_persistent_workers=False,\n",
      "dataloader_pin_memory=True,\n",
      "ddp_backend=None,\n",
      "ddp_broadcast_buffers=None,\n",
      "ddp_bucket_cap_mb=None,\n",
      "ddp_find_unused_parameters=None,\n",
      "ddp_timeout=1800,\n",
      "debug=[],\n",
      "deepspeed=None,\n",
      "disable_tqdm=False,\n",
      "dispatch_batches=None,\n",
      "do_eval=False,\n",
      "do_predict=False,\n",
      "do_train=False,\n",
      "eval_accumulation_steps=None,\n",
      "eval_delay=0,\n",
      "eval_steps=None,\n",
      "evaluation_strategy=IntervalStrategy.NO,\n",
      "fp16=False,\n",
      "fp16_backend=auto,\n",
      "fp16_full_eval=False,\n",
      "fp16_opt_level=O1,\n",
      "fsdp=[],\n",
      "fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},\n",
      "fsdp_min_num_params=0,\n",
      "fsdp_transformer_layer_cls_to_wrap=None,\n",
      "full_determinism=False,\n",
      "gradient_accumulation_steps=1,\n",
      "gradient_checkpointing=False,\n",
      "gradient_checkpointing_kwargs=None,\n",
      "greater_is_better=None,\n",
      "group_by_length=False,\n",
      "half_precision_backend=auto,\n",
      "hub_always_push=False,\n",
      "hub_model_id=None,\n",
      "hub_private_repo=False,\n",
      "hub_strategy=HubStrategy.EVERY_SAVE,\n",
      "hub_token=<HUB_TOKEN>,\n",
      "ignore_data_skip=False,\n",
      "include_inputs_for_metrics=False,\n",
      "include_num_input_tokens_seen=False,\n",
      "include_tokens_per_second=False,\n",
      "jit_mode_eval=False,\n",
      "label_names=None,\n",
      "label_smoothing_factor=0.0,\n",
      "learning_rate=5e-05,\n",
      "length_column_name=length,\n",
      "load_best_model_at_end=False,\n",
      "local_rank=0,\n",
      "log_level=passive,\n",
      "log_level_replica=warning,\n",
      "log_on_each_node=True,\n",
      "logging_dir=models/bert-base-cased-finetune-yelp/runs/Jan24_01-36-25_ecs-4325,\n",
      "logging_first_step=False,\n",
      "logging_nan_inf_filter=True,\n",
      "logging_steps=100,\n",
      "logging_strategy=IntervalStrategy.STEPS,\n",
      "lr_scheduler_kwargs={},\n",
      "lr_scheduler_type=SchedulerType.LINEAR,\n",
      "max_grad_norm=1.0,\n",
      "max_steps=-1,\n",
      "metric_for_best_model=None,\n",
      "mp_parameters=,\n",
      "neftune_noise_alpha=None,\n",
      "no_cuda=False,\n",
      "num_train_epochs=5,\n",
      "optim=OptimizerNames.ADAMW_TORCH,\n",
      "optim_args=None,\n",
      "output_dir=models/bert-base-cased-finetune-yelp,\n",
      "overwrite_output_dir=False,\n",
      "past_index=-1,\n",
      "per_device_eval_batch_size=8,\n",
      "per_device_train_batch_size=16,\n",
      "prediction_loss_only=False,\n",
      "push_to_hub=False,\n",
      "push_to_hub_model_id=None,\n",
      "push_to_hub_organization=None,\n",
      "push_to_hub_token=<PUSH_TO_HUB_TOKEN>,\n",
      "ray_scope=last,\n",
      "remove_unused_columns=True,\n",
      "report_to=[],\n",
      "resume_from_checkpoint=None,\n",
      "run_name=models/bert-base-cased-finetune-yelp,\n",
      "save_on_each_node=False,\n",
      "save_only_model=False,\n",
      "save_safetensors=True,\n",
      "save_steps=500,\n",
      "save_strategy=IntervalStrategy.STEPS,\n",
      "save_total_limit=None,\n",
      "seed=42,\n",
      "skip_memory_metrics=True,\n",
      "split_batches=False,\n",
      "tf32=None,\n",
      "torch_compile=False,\n",
      "torch_compile_backend=None,\n",
      "torch_compile_mode=None,\n",
      "torchdynamo=None,\n",
      "tpu_metrics_debug=False,\n",
      "tpu_num_cores=None,\n",
      "use_cpu=False,\n",
      "use_ipex=False,\n",
      "use_legacy_prediction_loop=False,\n",
      "use_mps_device=False,\n",
      "warmup_ratio=0.0,\n",
      "warmup_steps=0,\n",
      "weight_decay=0.0,\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# 完整的超参数配置\n",
    "print(training_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ebd3365-d359-4ab4-a300-4717590cc240",
   "metadata": {},
   "source": [
    "### 训练过程中的指标评估（Evaluate)\n",
    "\n",
    "**[Hugging Face Evaluate 库](https://huggingface.co/docs/evaluate/index)** 支持使用一行代码，获得数十种不同领域（自然语言处理、计算机视觉、强化学习等）的评估方法。 当前支持 **完整评估指标：https://huggingface.co/evaluate-metric**\n",
    "\n",
    "训练器（Trainer）在训练过程中不会自动评估模型性能。因此，我们需要向训练器传递一个函数来计算和报告指标。 \n",
    "\n",
    "Evaluate库提供了一个简单的准确率函数，您可以使用`evaluate.load`函数加载"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2a8ef138-5bf2-41e5-8c68-df8e11f4e98f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import evaluate\n",
    "\n",
    "metric = evaluate.load(\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d406c0-56d0-4a54-9c6c-e126ab7f5254",
   "metadata": {},
   "source": [
    "\n",
    "接着，调用 `compute` 函数来计算预测的准确率。\n",
    "\n",
    "在将预测传递给 compute 函数之前，我们需要将 logits 转换为预测值（**所有Transformers 模型都返回 logits**）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f46d2e59-1ebf-43d2-bc86-6b57a4d24d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "    logits, labels = eval_pred\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "    return metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2feba67-9ca9-4793-9a15-3eaa426df2a1",
   "metadata": {},
   "source": [
    "#### 训练过程指标监控\n",
    "\n",
    "通常，为了监控训练过程中的评估指标变化，我们可以在`TrainingArguments`指定`evaluation_strategy`参数，以便在 epoch 结束时报告评估指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "afaaee18-4986-4e39-8ad9-b8d413ab4cd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer\n",
    "\n",
    "training_args = TrainingArguments(output_dir=model_dir,\n",
    "                                  evaluation_strategy=\"epoch\", \n",
    "                                  per_device_train_batch_size=16,\n",
    "                                  num_train_epochs=3,\n",
    "                                  logging_steps=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47d6981-e444-4c0f-a7cb-dd7f2ba8df12",
   "metadata": {},
   "source": [
    "## 开始训练\n",
    "\n",
    "### 实例化训练器（Trainer）\n",
    "\n",
    "`kernel version` 版本问题：暂不影响本示例代码运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ca1d12ac-89dc-4c30-8282-f859724c0062",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=small_train_dataset,\n",
    "    eval_dataset=small_eval_dataset,\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a833e0db-1168-4a3c-8b75-bfdcef8c5157",
   "metadata": {},
   "source": [
    "## 使用 nvidia-smi 查看 GPU 使用\n",
    "\n",
    "为了实时查看GPU使用情况，可以使用 `watch` 指令实现轮询：`watch -n 1 nvidia-smi`:\n",
    "\n",
    "```shell\n",
    "Every 1.0s: nvidia-smi                                                   Wed Dec 20 14:37:41 2023\n",
    "\n",
    "Wed Dec 20 14:37:41 2023\n",
    "+---------------------------------------------------------------------------------------+\n",
    "| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |\n",
    "|-----------------------------------------+----------------------+----------------------+\n",
    "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
    "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
    "|                                         |                      |               MIG M. |\n",
    "|=========================================+======================+======================|\n",
    "|   0  Tesla T4                       Off | 00000000:00:0D.0 Off |                    0 |\n",
    "| N/A   64C    P0              69W /  70W |   6665MiB / 15360MiB |     98%      Default |\n",
    "|                                         |                      |                  N/A |\n",
    "+-----------------------------------------+----------------------+----------------------+\n",
    "\n",
    "+---------------------------------------------------------------------------------------+\n",
    "| Processes:                                                                            |\n",
    "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
    "|        ID   ID                                                             Usage      |\n",
    "|=======================================================================================|\n",
    "|    0   N/A  N/A     18395      C   /root/miniconda3/bin/python                6660MiB |\n",
    "+---------------------------------------------------------------------------------------+\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "accfe921-471d-481a-96da-c491cdebad0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='189' max='189' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [189/189 05:39, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.242100</td>\n",
       "      <td>1.090886</td>\n",
       "      <td>0.526000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.901400</td>\n",
       "      <td>0.960115</td>\n",
       "      <td>0.591000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.638200</td>\n",
       "      <td>0.978361</td>\n",
       "      <td>0.592000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=189, training_loss=0.9693943861300353, metrics={'train_runtime': 341.7098, 'train_samples_per_second': 8.779, 'train_steps_per_second': 0.553, 'total_flos': 789354427392000.0, 'train_loss': 0.9693943861300353, 'epoch': 3.0})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6d581099-37a4-4470-b051-1ada38554089",
   "metadata": {},
   "outputs": [],
   "source": [
    "small_test_dataset = tokenized_datasets[\"test\"].shuffle(seed=64).select(range(100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ffb47eab-1370-491e-8a84-6d5347a350b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='13' max='13' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [13/13 00:02]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 1.0753791332244873,\n",
       " 'eval_accuracy': 0.52,\n",
       " 'eval_runtime': 2.9889,\n",
       " 'eval_samples_per_second': 33.457,\n",
       " 'eval_steps_per_second': 4.349,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate(small_test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27a55686-7c43-4ab8-a5cd-0e77f14c7c52",
   "metadata": {},
   "source": [
    "### 保存模型和训练状态\n",
    "\n",
    "- 使用 `trainer.save_model` 方法保存模型，后续可以通过 from_pretrained() 方法重新加载\n",
    "- 使用 `trainer.save_state` 方法保存训练状态"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ad0cbc14-9ef7-450f-a1a3-4f92b6486f41",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.save_model(model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f6e30510-0536-49d4-8e1b-43fc25272bde",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "badf5868-2847-439d-a73e-42d1cca67b5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.save_state()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c9441ad-f65a-42b7-9016-4809c78285e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fd92e35d-fed7-4ff2-aa84-27b5e29b917a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainer.model.save_pretrained(\"./\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61828934-01da-4fc3-9e75-8d754c25dfbc",
   "metadata": {},
   "source": [
    "## Homework: 使用完整的 YelpReviewFull 数据集训练，看 Acc 最高能到多少"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ee2580a-7a5a-46ae-a28b-b41e9e838eb1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
